DCT-Former: Efficient Self-Attention with Discrete Cosine Transform

نویسندگان

چکیده

Since their introduction the Trasformer architectures emerged as dominating for both natural language processing and, more recently, computer vision applications. An intrinsic limitation of this family "fully-attentive" arises from computation dot-product attention, which grows in memory consumption and number operations $O(n^2)$ where $n$ stands input sequence length, thus limiting applications that require modeling very long sequences. Several approaches have been proposed so far literature to mitigate issue, with varying degrees success. Our idea takes inspiration world lossy data compression (such JPEG algorithm) derive an approximation attention module by leveraging properties Discrete Cosine Transform. extensive section experiments shows our method up less same performance, while also drastically reducing inference time. This makes it particularly suitable real-time contexts on embedded platforms. Moreover, we assume results research might serve a starting point broader deep neural models reduced footprint. The implementation will be made publicly available at https://github.com/cscribano/DCT-Former-Public

برای دانلود باید عضویت طلایی داشته باشید

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Image Steganography Using Discrete Cosine Transform (DCT) and Blowfish Algorithm

Steganography is one of the methods of secret communication that hides the existence of message so that a viewer cannot detect the transmission of message and hence cannot try to decrypt it. It is the process of embedding secret data in the cover image without significant changes to the cover image. A cryptography algorithm is used to convert the secret messages to an unreadable form before emb...

متن کامل

The Discrete Cosine Transform (DCT): Theory and Application

Transform coding constitutes an integral component of contemporary image/video processing applications. Transform coding relies on the premise that pixels in an image exhibit a certain level of correlation with their neighboring pixels. Similarly in a video transmission system, adjacent pixels in consecutive frames 2 show very high correlation. Consequently, these correlations can be exploited ...

متن کامل

Energy-Efficient Discrete Cosine Transform on FPGAs

The 2-D discrete cosine transform (DCT) is an integral part of video and image processing; it is used in both the JPEG and MPEG encoding standards. As streaming video is brought to mobile devices, it becomes important that it is possible to calculate the DCT in an energy-efficient manner. In this paper, we present a new algorithm and processing element (PE) architecture for computing the DCT wi...

متن کامل

JPEG Encoder using Discrete Cosine Transform & Inverse Discrete Cosine Transform

In the past decade, the advancement in data communications was significant during explosive growth of the Internet, which led to the demand for using multimedia in portable devices. Video and Audio data streams require a huge amount of bandwidth to be transferred in an uncompressed form. The objective of this paper is to minimize the number of bits required to represent an image and also the ac...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

ژورنال

عنوان ژورنال: Journal of Scientific Computing

سال: 2023

ISSN: ['1573-7691', '0885-7474']

DOI: https://doi.org/10.1007/s10915-023-02125-5